基于生成模型的运动预测技术最近已经实现了预测受控人类运动的，例如预测具有相似下体运动的多个上层运动。但是，为了实现这一目标，最新的方法要求随后学习映射功能以寻求类似的动作或重复训练模型以控制身体的所需部分。在本文中，我们提出了一个新颖的框架，以学习可控人体运动预测的脱节表示。我们的网络涉及有条件的变分自动编码器（CVAE）结构，以模拟全身人体运动，以及仅学习相应的部分体体（例如，下体）运动的额外CVAE路径。具体而言，额外CVAE路径施加的电感偏置鼓励两个路径中的两个潜在变量分别控制每个部分运动运动的单独表示。通过一次训练，我们的模型能够为生成的人类动作提供两种类型的控制：（i）严格控制人体的一部分，（ii）通过从一对潜在空间中取样来自适应控制另一部分。此外，我们将抽样策略扩展到了我们训练的模型，以多样化可控的预测。我们的框架还可以通过灵活地自定义额外CVAE路径的输入来允许新的控制形式。广泛的实验结果和消融研究表明，我们的方法能够在质量和定量上预测最新的可控人体运动。

在本文中，我们介绍了一种基于生成的模型的方法来产生多样化的人类运动插值结果。我们通过利用编码器和解码器的经常性神经网络（RNN）结构来求助于在一对给定的开始和结束动作上学习人为变形自动编码器（CVAE）。此外，我们介绍了正规化损失，以进一步促进样本多样性。曾经训练过，我们的方法能够通过从学习的潜在空间重复地抽样来产生多种合理的相干动作。在公开的数据集上实验证明了我们方法的有效性，在样本合理性和多样性方面。

在计算机视觉中起关键作用的人类运动预测通常需要过去的运动序列作为输入。但是，在实际应用中，完整而正确的过去运动顺序可能太贵了。在本文中，我们提出了一种新的方法，可以从更弱的条件（即单个图像）中预测未来的人类运动，并具有混合密度网络（MDN）建模。与大多数现有的深层人类运动预测方法相反，MDN的多模式性质可以产生各种未来的运动假设，这很好地补偿了由单个输入和人类运动不确定性汇总的强烈随机歧义。在设计损失函数时，我们进一步引入了基于能量的公式，以灵活地对MDN的可学习参数施加先前的损失，以保持运动相干性，并通过自定义能量功能来提高预测准确性。我们训练有素的模型将图像直接作为输入，并生成满足给定条件的多个合理动作。在两个标准基准数据集上进行的广泛实验证明了我们方法在预测多样性和准确性方面的有效性。

在图像编辑中起着关键作用的颜色传输，最近引起了明显的关注。由于各种问题，诸如耗时的手动调整和前提分割问题等各种问题，迄今为止仍然是挑战。在本文中，我们建议在概率框架下模拟颜色转移并将其投射为参数估计问题。特别地，我们在高斯混合模型（GMM）下将传送图像与示例图像相关联，并将转移的图像颜色视为GMM质心。我们采用期望 - 最大化（EM）算法（E-Step和M步）进行优化。为了更好地保留梯度信息，我们将基于LAPPRIAN的正则化术语引入到通过导出梯度下降算法来解决的M-Tep的目标函数。鉴于源图像和示例图像的输入，我们的方法能够随着EM迭代的增加来生成连续的颜色传输结果。各种实验表明，我们的方法通常优于视觉和定量的其他竞争色彩转移方法。

Large training data and expensive model tweaking are standard features of deep learning for images. As a result, data owners often utilize cloud resources to develop large-scale complex models, which raises privacy concerns. Existing solutions are either too expensive to be practical or do not sufficiently protect the confidentiality of data and models. In this paper, we study and compare novel \emph{image disguising} mechanisms, DisguisedNets and InstaHide, aiming to achieve a better trade-off among the level of protection for outsourced DNN model training, the expenses, and the utility of data. DisguisedNets are novel combinations of image blocktization, block-level random permutation, and two block-level secure transformations: random multidimensional projection (RMT) and AES pixel-level encryption (AES). InstaHide is an image mixup and random pixel flipping technique \cite{huang20}. We have analyzed and evaluated them under a multi-level threat model. RMT provides a better security guarantee than InstaHide, under the Level-1 adversarial knowledge with well-preserved model quality. In contrast, AES provides a security guarantee under the Level-2 adversarial knowledge, but it may affect model quality more. The unique features of image disguising also help us to protect models from model-targeted attacks. We have done an extensive experimental evaluation to understand how these methods work in different settings for different datasets.

A storyboard is a roadmap for video creation which consists of shot-by-shot images to visualize key plots in a text synopsis. Creating video storyboards however remains challenging which not only requires association between high-level texts and images, but also demands for long-term reasoning to make transitions smooth across shots. In this paper, we propose a new task called Text synopsis to Video Storyboard (TeViS) which aims to retrieve an ordered sequence of images to visualize the text synopsis. We construct a MovieNet-TeViS benchmark based on the public MovieNet dataset. It contains 10K text synopses each paired with keyframes that are manually selected from corresponding movies by considering both relevance and cinematic coherence. We also present an encoder-decoder baseline for the task. The model uses a pretrained vision-and-language model to improve high-level text-image matching. To improve coherence in long-term shots, we further propose to pre-train the decoder on large-scale movie frames without text. Experimental results demonstrate that our proposed model significantly outperforms other models to create text-relevant and coherent storyboards. Nevertheless, there is still a large gap compared to human performance suggesting room for promising future work.

Solving real-world optimal control problems are challenging tasks, as the system dynamics can be highly non-linear or including nonconvex objectives and constraints, while in some cases the dynamics are unknown, making it hard to numerically solve the optimal control actions. To deal with such modeling and computation challenges, in this paper, we integrate Neural Networks with the Pontryagin's Minimum Principle (PMP), and propose a computationally efficient framework NN-PMP. The resulting controller can be implemented for systems with unknown and complex dynamics. It can not only utilize the accurate surrogate models parameterized by neural networks, but also efficiently recover the optimality conditions along with the optimal action sequences via PMP conditions. A toy example on a nonlinear Martian Base operation along with a real-world lossy energy storage arbitrage example demonstrates our proposed NN-PMP is a general and versatile computation tool for finding optimal solutions. Compared with solutions provided by the numerical optimization solver with approximated linear dynamics, NN-PMP achieves more efficient system modeling and higher performance in terms of control objectives.

The task of reconstructing 3D human motion has wideranging applications. The gold standard Motion capture (MoCap) systems are accurate but inaccessible to the general public due to their cost, hardware and space constraints. In contrast, monocular human mesh recovery (HMR) methods are much more accessible than MoCap as they take single-view videos as inputs. Replacing the multi-view Mo- Cap systems with a monocular HMR method would break the current barriers to collecting accurate 3D motion thus making exciting applications like motion analysis and motiondriven animation accessible to the general public. However, performance of existing HMR methods degrade when the video contains challenging and dynamic motion that is not in existing MoCap datasets used for training. This reduces its appeal as dynamic motion is frequently the target in 3D motion recovery in the aforementioned applications. Our study aims to bridge the gap between monocular HMR and multi-view MoCap systems by leveraging information shared across multiple video instances of the same action. We introduce the Neural Motion (NeMo) field. It is optimized to represent the underlying 3D motions across a set of videos of the same action. Empirically, we show that NeMo can recover 3D motion in sports using videos from the Penn Action dataset, where NeMo outperforms existing HMR methods in terms of 2D keypoint detection. To further validate NeMo using 3D metrics, we collected a small MoCap dataset mimicking actions in Penn Action,and show that NeMo achieves better 3D reconstruction compared to various baselines.

A major goal of multimodal research is to improve machine understanding of images and text. Tasks include image captioning, text-to-image generation, and vision-language representation learning. So far, research has focused on the relationships between images and text. For example, captioning models attempt to understand the semantics of images which are then transformed into text. An important question is: which annotation reflects best a deep understanding of image content? Similarly, given a text, what is the best image that can present the semantics of the text? In this work, we argue that the best text or caption for a given image is the text which would generate the image which is the most similar to that image. Likewise, the best image for a given text is the image that results in the caption which is best aligned with the original text. To this end, we propose a unified framework that includes both a text-to-image generative model and an image-to-text generative model. Extensive experiments validate our approach.

Model-based attacks can infer training data information from deep neural network models. These attacks heavily depend on the attacker's knowledge of the application domain, e.g., using it to determine the auxiliary data for model-inversion attacks. However, attackers may not know what the model is used for in practice. We propose a generative adversarial network (GAN) based method to explore likely or similar domains of a target model -- the model domain inference (MDI) attack. For a given target (classification) model, we assume that the attacker knows nothing but the input and output formats and can use the model to derive the prediction for any input in the desired form. Our basic idea is to use the target model to affect a GAN training process for a candidate domain's dataset that is easy to obtain. We find that the target model may distract the training procedure less if the domain is more similar to the target domain. We then measure the distraction level with the distance between GAN-generated datasets, which can be used to rank candidate domains for the target model. Our experiments show that the auxiliary dataset from an MDI top-ranked domain can effectively boost the result of model-inversion attacks.